Scalable low-latency mesh interconnect for switch chips

ABSTRACT

A device implementing a scalable low-latency mesh may include a memory management unit, an egress processor, and an egress cell circuit that includes at least a first queue and a second queue. The memory management unit may be configured to buffer first cells for transmission. The egress cell circuit may be configured to queue the first cells from the memory management unit in the first queue, queue second cells from an off-chip memory management unit of another device in the second queue, and schedule the first cells from the first queue and second cells from the second queue for transmission via an egress processor. The egress processor may be configured to transmit the first and second cells over at least one first port.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/293,287, entitled “Scalable Low-Latency MeshInterconnect for Switch Chips,” filed on Feb. 9, 2016, which is herebyincorporated by reference in its entirety for all purposes.

TECHNICAL FIELD

The present description relates generally to a scalable low-latencymesh, including a scalable low latency mesh interconnect for switchchips of a switch device.

BACKGROUND

In the enterprise switch market, bandwidth per switch chip is doublingevery eighteen months. As a result of the bandwidth increase, powerdensity is also increasing while cost is decreasing. In order to supportthe enterprise switch market, significant cooling technology is requiredto support the continuing growth with the single system on a chip (SoC)implementations. However, an efficient way to scale a system usingmultiple switch chips may allow for a reduction in the cooling and/orpower requirements.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain features of the subject technology are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of thesubject technology are set forth in the following figures.

FIG. 1 illustrates an example network environment in which a switchdevice implementing a scalable low-latency mesh interconnect for switchchips may be implemented in accordance with one or more implementations.

FIG. 2 illustrates an example switch device implementing a scalablelow-latency mesh interconnect for two switch chips in accordance withone or more implementations.

FIG. 3 illustrates an example switch device implementing a scalablelow-latency mesh interconnect for three switch chips in accordance withone or more implementations.

FIG. 4 illustrates an example transit first-in-first-out (fifo) of anexample switch chip of a switch device implementing a scalablelow-latency mesh interconnect for switch chips in accordance with one ormore implementations.

FIG. 5 illustrates an example egress cell fifo of an example switch chipof a switch device implementing a scalable low-latency mesh interconnectfor switch chips in accordance with one or more implementations.

FIG. 6 illustrates a flow diagram of an example process of a transitfifo of an example switch chip of a switch device implementing ascalable low-latency mesh interconnect for switch chips in accordancewith one or more implementations.

FIG. 7 illustrates a flow diagram of an example process of an egresscell fifo of an example switch chip of a switch device implementing ascalable low-latency mesh interconnect for switch chips in accordancewith one or more implementations.

FIG. 8 conceptually illustrates an example electronic system with whichone or more implementations of the subject technology can beimplemented.

DETAILED DESCRIPTION

The detailed description set forth below is intended as a description ofvarious configurations of the subject technology and is not intended torepresent the only configurations in which the subject technology may bepracticed. The appended drawings are incorporated herein and constitutea part of the detailed description. The detailed description includesspecific details for the purpose of providing a thorough understandingof the subject technology. However, the subject technology is notlimited to the specific details set forth herein and may be practicedusing one or more implementations. In one or more instances, structuresand components are shown in block diagram form in order to avoidobscuring the concepts of the subject technology.

FIG. 1 illustrates an example network environment 100 in which a switchdevice 110 implementing a scalable low-latency mesh interconnect forswitch chips may be implemented in accordance with one or moreimplementations. Not all of the depicted components may be required,however, and one or more implementations may include additionalcomponents not shown in the figure. Variations in the arrangement andtype of the components may be made without departing from the spirit orscope of the claims as set forth herein. Additional components,different components, or fewer components may be provided.

The example network environment 100 includes various electronic devices102, 104, 106, such as one or more servers, computers, data storagedevices, network components, network devices, etc. In one or moreimplementations, the network environment 100 includes a set of servers,a server bank, a storage area network, and/or a vast network ofinterconnected computers or network devices. In one or moreimplementations, the network environment 100 includes one or more otherdevices, such as, for example, one or more mobile devices, tabletdevices, set-top box (STB) devices, storage devices, desktop computers,routers, switches, bridges, or any other machine or device.

One or more network devices, such as the switch device 110, may beimplemented in the network environment 100 to facilitate communicationbetween the one or more electronic devices 102, 104, 106. The electronicdevices 102, 104, 106 of the network environment 100, are connected orotherwise in communication with each other, through or using the switchdevice 110. For example, the electronic devices 102, 104, 106 may eachbe coupled to one or more physical ports of the switch device 110 by anetwork transmission line, such as an Ethernet transmission line, acoaxial transmission line, an optical transmission line, fibre channel,or generally any transmission line. In one or more implementations, oneor more of the electronic devices 102, 104, 106 is wirelessly coupled tothe switch device 110. The switch device 110 and/or one or more of theelectronic devices 102, 104, 106 may be and/or may include all or partof the electronic system discussed below with respect to FIG. 8.

The switch device 110 may include a number of integrated circuits, orchips, that are referred to as switch chips. The switch chips may bemounted on a common substrate and/or one or more of the switch chips maybe mounted on separate substrates, such as separate line cards. Eachswitch chip is coupled to one or more ports of the switch device 110 andprocesses data units transmitted through and/or received over, theports. For explanatory purposes, the data units may be primarilyreferred to herein as cells, e.g. portions of packets; however, the dataunits may be packets, protocol data units (PDUs), chunks, or generallyany delineable data unit. To facilitate processing the data units, eachof the switch chips may include one or more ingress processors, egressprocessors, and/or memory management units (MMUs).

In the subject system of a scalable low-latency mesh interconnect forswitch chips, the switch chips are interconnected within the switchdevice 110 using a low latency mesh network. The switch chips may beconnected such that, at most, a single store and forward operationoccurs for any data unit routed through the switch device 110.Furthermore, each data unit may only pass through, at most, a singleingress processor and a single egress processor, across the entiremulti-chip architecture, thereby incurring a single forwarding latencyfor each data unit. In one or more implementations, the onboard meshconnection of the switch chips may include a serializer/deserializer(SerDes) interface running at a high speed. In one example, each switchchip's packet processor may be individually capable of a particularbandwidth. However, when the switch chips are interconnected in theswitch device 110 using the subject system, the packet processors of thechips may run at a fraction of their full capacity, thereby reducing thepower consumption of the chips and consequently the switch device. Anexample switch device 110 implementing the subject system is discussedfurther below with respect to FIGS. 2 and 3.

In one or more implementations, a fabric implementing the subject systemmay carry a system header with each data unit, such as each packet, inorder to facilitate implementing the subject system. The fabric may bebyte packed, e.g. the Ethernet interpacket gap may be used to pack thesystem header. The header may have a variable size, such as a first sizefor basic unicast flows and a second size when features such asinstrumentation are turned on. The first size may match the interpacketgap thereby not requiring any overspeed.

FIG. 2 illustrates an example switch device 110 implementing a scalablelow-latency mesh interconnect for two switch chips 210A-B in accordancewith one or more implementations. Not all of the depicted components maybe used, however, and one or more implementations may include additionalcomponents not shown in the figure. Variations in the arrangement andtype of the components may be made without departing from the spirit orscope of the claims as set forth herein. Additional components,different components, or fewer components may be provided. Furthermore,one or more of the illustrated components may be merged/combined into asingle component and/or one or more of the illustrated components may besplit into multiple separate components.

The switch device 110 includes a first switch chip 210A and a secondswitch chip 210B that are interconnected by links 211A-B. The firstswitch chip 210A includes a transit fifo 212A, a memory management unit(MMU) 214A, an egress cell fifo 215A, an ingress processor 216A, a portmacro 217A, and an egress processor 218A. The second switch chip 210Bmay include a transit fifo 212B, an MMU 214B, an egress cell fifo 215B,an ingress processor 216B, a port macro 217B, and an egress processor218B.

The ingress processors 216A-B may be, for example, ingress packetprocessors, and the egress processors 218A-B may be, for example, egresspacket processors. The port macros 217A-B may each be coupled toseparate sets of one or more ports. For example, the electronic device102 of FIG. 1 may be connected to a port coupled to the port macro 217Aof the switch chip 210A, and the electronic device 104 of FIG. 1 may beconnected to a port coupled to the port macro 217B of the switch chip210B. The links 211A-B may each be separate physical connections, e.g.on a substrate, that connect the switch chips 210A-B. The links 211A-Bmay each be bidirectional or unidirectional.

The MMUs 214A-B of each switch chip 210A-B may each include virtualoutput queues corresponding to each port coupled to each switch chip210A-B. The egress cell fifos 215A-B of each of the switch chips 210A-Bincludes output queues for all of the ports in the switch device 110,such as 256 ports, or any number of ports. The switch chips 210A-Butilize the transit fifos 212A-B to support the mesh network topology.The transit fifos 212A-B each drives a serializer-deserializer interfacefor the mesh network connections via the links 211A-B. The transit fifos212A-B may each implement cut-through switching by looking at the headerto either copy a cell, drop a cell, or pass through the cell. Thetransit fifos 212A-B may also be referred to as transit circuits. Anexample transit fifo 212A is discussed further below with respect toFIG. 4, and an example process of an example transit fifo 212A isdiscussed further below with respect to FIG. 6.

The switch chips 210A-B also utilize the egress cell fifos 215A-B topull/receive cells from the respective MMUs 214A-B and the respectivetransit fifos 212A-B as input. The egress cell fifos 215A-B may becell-based, packet-based, or generally any data unit based. The egresscell fifos 215A-B may also be referred to as egress cell circuits. Anexample egress cell fifo 215A is discussed further below with respect toFIG. 5, and an example process of an example egress cell fifo 215A isdiscussed further below with respect to FIG. 7.

In operation, the ingress processor 216A receives cells from the coupledports via the port macro 217A. The ingress processor 216A stores thecells for transmission in the MMU 214A. In one or more implementations,for local cut-through switching, the ingress processor 216A provides thecells directly to the egress cell fifo 215A or the egress processor218A. In one or more implementations, the local cut-through switchingmay be dependent on the occupancy of the queues of the egress cell fifo215A. The egress cell fifo 215A retrieves cells from the on-chip MMU214A and queues the cells in dedicated fifos (or queues/buffers), fortransmission over the coupled ports via the egress processor 218A andthe port macro 217A. The components of the switch chip 210A may bereferred to as on-chip (relative to the egress cell fifo 215A), and thecomponents of the switch chip 210B (or any other chip), may be referredto as off-chip (with respect to the egress cell fifo 215A).

The egress cell fifo 215A also retrieves cells from the off-chip MMU214B of the switch chip 210B and queues the cells in separate dedicatedfifos for transmission over the coupled ports via the egress processor218A and the port macro 217A. For example, the egress cell fifo 215Arequests cells from the off-chip MMU 214B via the transit fifo 212A, thelink 211A, and the transit fifo 212B. The transit fifo 212A retrievesthe cells from the off-chip MMU 214B via the transit fifo 212B (withoutthe cells passing through the egress processor 218B of the switch chip210B), and the transit fifo 212A provides the cells directly to theegress cell fifo 215A, e.g. bypassing the on-chip MMU 214A. The egresscell fifo 215A then queues the cells retrieved from the off-chip MMU214B for transmission over coupled ports via the egress processor 218Aand the port macro 217A, e.g. dispersed within the transmissions of thecells retrieved from the on-chip MMU 214A. Thus, some of the cellsretrieved from the off-chip MMU 214B are initially received via theingress processor 216B of the switch chip 210B, but are not transmittedvia the egress processor 218B of the switch chip 210B, but rather theegress processor 218A of the switch chip 210A. The egress cell fifo 215Bof the switch chip 210B may operate in a similar manner.

The switch device 110 may perform multicast replication at the one ofthe MMUs 214A-B of the one of the switch chips 210A-B that is the sourceof the multicast. The switch device 110 may also utilize priority flowcontrol to prevent draining from the egress cell fifos 215A-B that areorganized per queue, and credit control may utilized to prevent drainingfrom the source MMUs 214A-B.

The switch device 110 may also utilize metering on equal cost multipath(ECMP) and/or link aggregation protocol (LAG) channels within each ofthe switch chips 210A-B and/or for ports across the switch chips 210A-B.For example, a slow update may be utilized to synchronize the meterstate across the mesh network for both of the switch chips 210A-B. Theupdate rate can be low when the number of switch chips 210A-B is low.

In one or more implementations, one or more of the switch chips 210A-B,the transit fifos 212A-B, the MMUs 214A-B, the egress cell fifos 215A-B,the ingress processors 216A-B, the port macros 217A-B, the egressprocessors 218A-B, and/or the links 211A-B may be implemented insoftware, and/or may be implemented in hardware (e.g., an ApplicationSpecific Integrated Circuit (ASIC), a Field Programmable Gate Array(FPGA), a Programmable Logic Device (PLD), a controller, a statemachine, gated logic, discrete hardware components, or any othersuitable devices) and/or may be implemented in a combination of both.Additional features and functions of these modules according to variousaspects of the subject technology are further described in the presentdisclosure.

FIG. 3 illustrates an example switch device 110 implementing a scalablelow-latency mesh interconnect for three switch chips 210A-B, 310 inaccordance with one or more implementations. Not all of the depictedcomponents may be used, however, and one or more implementations mayinclude additional components not shown in the figure. Variations in thearrangement and type of the components may be made without departingfrom the spirit or scope of the claims as set forth herein. Additionalcomponents, different components, or fewer components may be provided.Furthermore, one or more of the illustrated components may bemerged/combined into a single component and/or one or more of theillustrated components may be split into multiple separate components.

The switch device 110 includes the switch chips 210A-B of FIG. 2 as wellas a third switch chip 310. The switch chip 310 includes a transit fifo312, an MMU 314, an egress cell fifo 315, an ingress processor 316, aport macro 317, and an egress processor 318. The switch chips 210A-B,310 are interconnected via links 311A-C. The links 311A-C may each beunidirectional or bidirectional. Thus, as shown in FIG. 3, the switchchip 210A can communicate directly with the switch chip 310 via the link311A, and the switch chip 210A can communicate indirectly with theswitch chip 310, e.g. using the switch chip 210B as an intermediary, viathe links 311B-C.

In order to support the multiple different links 311A-C, the transitfifos 212A-B, 312 may each include a scheduler, as is discussed furtherbelow with respect to FIG. 4. The schedulers of the transit fifos212A-B, 312 may track link occupancy and may share link stateinformation, such as link and path utilization statistics, with eachother. The schedulers of the individual transit fifos 212A-B, 312 mayuse the link state information to determine the appropriate link totransmit a given cell over, e.g. to ensure that the links 311A-C arebalanced on a cell-by-cell basis. Thus, the transit fifo 212A may selectthe link 311A or the links 311B-C for transmitting a given cell to thetransit fifo 312, and the transit fifo 212A may select the link 311B orthe links 311A,C for transmitting a given cell to the transit fifo 212B.The transit fifos 212B, 312 may operate in a similar manner. The transitfifos 212A-B, 312 may determine whether to copy, pass-thru, or delete agiven cell based at least in part on the system header associated withthe cell.

The subject system may also be utilized in a configuration planeimplementation where the switch chips 210A-B, 310 are presented as threeendpoints of a memory map, such as three peripheral componentinterconnect express (PCIe or PCI-E) endpoints. In this instance, theswitch chips 210A-B, 310 may be coupled to a PCIe riser card which iscoupled to a memory module and a host CPU. One of the switch chips210A-B, 310 is selected or elected to perform the packet punt path forthe implementation. The CPU queues can be load balanced and a givenqueue can be associated with a single one of the switch chips 210A-B,310.

In one or more implementations, one or more of the switch chips 210A-B,310, the transit fifos 212A-B, 312, the MMUs 214A-B, 314, the egresscell fifos 215A-B, 315, the ingress processors 216A-B, 316, the portmacros 217A-B, 317, the egress processors 218A-B, 318, and/or the links311A-C may be implemented in software, and/or may be implemented inhardware (e.g., an Application Specific Integrated Circuit (ASIC), aField Programmable Gate Array (FPGA), a Programmable Logic Device (PLD),a controller, a state machine, gated logic, discrete hardwarecomponents, or any other suitable devices) and/or may be implemented ina combination of both. Additional features and functions of thesemodules according to various aspects of the subject technology arefurther described in the present disclosure.

FIG. 4 illustrates an example transit fifo 212A of an example switchchip 210A of a switch device 110 implementing a scalable low-latencymesh interconnect for switch chips in accordance with one or moreimplementations. Not all of the depicted components may be used,however, and one or more implementations may include additionalcomponents not shown in the figure. Variations in the arrangement andtype of the components may be made without departing from the spirit orscope of the claims as set forth herein. Additional components,different components, or fewer components may be provided. Furthermore,one or more of the illustrated components may be merged/combined into asingle component and/or one or more of the illustrated components may besplit into multiple separate components.

The example transit fifo 212A includes a scheduler 410 for schedulingcells for transmission over the links 311A-B, and one or more buffers420 for buffering cells received over the links 311A-B. The one or morebuffers 420 may be a small buffer that is utilized to latency match,e.g. account for latency between the cells received over the differentlinks 311A-B, and to keep received cells in order.

In operation, the transit fifo 212A may receive a request for cellsstored in the MMU 214A. The request may be received from, for example,the transit fifo 212B on behalf of the egress cell fifo 215B and/or thetransit fifo 312 on behalf of the egress cell fifo 315. The transit fifo212A schedules the requested cells for transmission over one of thelinks 311A-B, retrieves the requested cells from the MMU 214A at thescheduled time and transmits the cells to the transit fifo 212B and/orthe transit fifo 312 via a selected one of the links 311A-B. The transitfifo 212A may receive link state information from the transit fifos212B, 312 and may provide link state information to the transit fifos212B, 312. The scheduler 410 determines the appropriate link fortransmitting each cell based at least in part on the received link stateinformation.

The transit fifo 212A also requests cells from the MMUs 214B, 314 onbehalf of the egress cell fifo 215A. The requested cells are receivedover one or more of the links 311A-B and are buffered in the one or morebuffers 420. Once the cells have been reordered, if necessary, thetransit fifo 212A transmit the cells directly to the egress cell fifo215A, bypassing the MMU 214A, thereby minimizing the number ofread/write operations for processing the cells and consequentlydecreasing the latency associated with processing the cells.

In one or more implementations, one or more of the switch chip 210A, thetransit fifo 212A, the scheduler 410, the buffer 420, and/or the links311A-B may be implemented in software, and/or may be implemented inhardware (e.g., an Application Specific Integrated Circuit (ASIC), aField Programmable Gate Array (FPGA), a Programmable Logic Device (PLD),a controller, a state machine, gated logic, discrete hardwarecomponents, or any other suitable devices) and/or may be implemented ina combination of both. Additional features and functions of thesemodules according to various aspects of the subject technology arefurther described in the present disclosure.

FIG. 5 illustrates an example egress cell fifo 215A of an example switchchip 210A of a switch device 110 implementing a scalable low-latencymesh interconnect for switch chips in accordance with one or moreimplementations. Not all of the depicted components may be used,however, and one or more implementations may include additionalcomponents not shown in the figure. Variations in the arrangement andtype of the components may be made without departing from the spirit orscope of the claims as set forth herein. Additional components,different components, or fewer components may be provided. Furthermore,one or more of the illustrated components may be merged/combined into asingle component and/or one or more of the illustrated components may besplit into multiple separate components.

The egress cell fifo 215A includes one or more queues 505A-D and ascheduler 510. For example, the egress cell fifo 215A may include aseparate dedicated queues for each output queue of each port of theswitch device 110, e.g. across each of the switch chips of the switchdevice 110. Thus, if the switch device 110 includes three switch chips212A-B, 312, each of the switch chips 212A-B, 312 is coupled to 64ports, and each port is associated with 4 output queues, then the egresscell fifo 215A includes 64*4*3=768 queues. Thus, when the egress cellfifo 215A receives cells from one of the off-chip MMUs 214B, 314, theegress cell fifo 215A queues the cells in the appropriate one of thequeues 505A-D associated with the one of the off-chip MMUs 214B, 314.

In operation, the egress cell fifo 215A retrieves cells from a read portof the on-chip MMU 214A, and from the off-chip MMUs 214A, 314, via thetransit fifo 212A. The egress cell fifo 215A retrieves cells from therespective MMUs 214A-B, 314 as space allows in the queues 505A-Dassociated with the respective MMUs 214A-B, 314. The scheduler 510schedules output queues to a given coupled port using, e.g., sourcebased credit for each port. The scheduler 510 ensures that the cellsfrom the on-chip MMU 214A and the cells from the off-chip MMUs 214B, 314are given fair access to a given port.

In one or more implementations, one or more of the switch chip 210A, theegress cell fifo 215A, the queues 505A-D, and/or the scheduler 510 maybe implemented in software, and/or may be implemented in hardware (e.g.,an Application Specific Integrated Circuit (ASIC), a Field ProgrammableGate Array (FPGA), a Programmable Logic Device (PLD), a controller, astate machine, gated logic, discrete hardware components, or any othersuitable devices) and/or may be implemented in a combination of both.Additional features and functions of these modules according to variousaspects of the subject technology are further described in the presentdisclosure.

FIG. 6 illustrates a flow diagram of an example process 600 of a transitfifo 212A of an example switch chip 210A of a switch device 110implementing a scalable low-latency mesh interconnect for switch chipsin accordance with one or more implementations. For explanatorypurposes, the example process 600 is primarily described herein withreference to the transit fifo 212A of the switch chip 210A of the switchdevice 110 of FIGS. 2-3; however, the example process 600 is not limitedto the transit fifo 212A of the switch chip 210A of FIGS. 2-3, e.g. theexample process 600 may be performed by one or more of the transit fifos212B, 312 of the switch chips 210B, 310, and/or the example process 600may be performed by one or more components of the transit fifo 212A.Further for explanatory purposes, the blocks of the example process 600are described herein as occurring in serial, or linearly. However,multiple blocks of the example process 600 may occur in parallel. Inaddition, the blocks of the example process 600 may be performed adifferent order than the order shown and/or one or more of the blocks ofthe example process 600 may not be performed.

The transit fifo 212A receives a request from an off-chip transit fifo212B for cells stored in the on-chip MMU 214A (602). The transit fifo212A determines an appropriate link to transmit the cells to theoff-chip transit fifo 212B (604). For example, if there are multiplelinks 311A-C for transmitting cells to the off-chip transit fifo 212B,the transit fifo 212A determines the appropriate link based on linkstate information, such as link occupancy information, received from theoff-chip transit fifo 212B and/or the off-chip transit fifo 312. Thetransit fifo 212A retrieves the requested cells directly from the readport of the MMU 214A (606) and transmits the cells via the determinedlink to the off-chip transit fifo 212B for transmission via an off-chipegress processor 218B (608). In this manner, the cells retrieved fromthe MMU 214A bypass, e.g. are not processed by, the on-chip egressprocessor 218A.

The transit fifo 212A receives a request from the on-chip egress cellfifo 215A for cells stored in an off-chip MMU, such as the off-chip MMU214B (610). The transit fifo 212A transmits the request to the off-chiptransit fifo 212B that is on the same switch chip 212B as the off-chipMMU 214B (612). The transit fifo 212A receives the cells from theoff-chip MMU 214B from the off-chip transit fifo 212B (614). The transitfifo 212A transmits the received cells to the on-chip egress cell fifo215A for transmission via the on-chip egress processor 218A (616).

FIG. 7 illustrates a flow diagram of an example process 700 of an egresscell fifo 215A of an example switch chip 210A of a switch device 110implementing a scalable low-latency mesh interconnect for switch chipsin accordance with one or more implementations. For explanatorypurposes, the example process 700 is primarily described herein withreference to the egress cell fifo 215A of the switch chip 210A of theswitch device 110 of FIGS. 2-3; however, the example process 700 is notlimited to the egress cell fifo 215A of the switch chip 210A of FIGS.2-3, e.g. the example process 700 may be performed by one or more of theegress cell fifos 215A, 315 of the switch chips 210B, 310, and/or theexample process 700 may be performed by one or more components of theegress cell fifo 215A. Further for explanatory purposes, the blocks ofthe example process 700 are described herein as occurring in serial, orlinearly. However, multiple blocks of the example process 700 may occurin parallel. In addition, the blocks of the example process 700 may beperformed a different order than the order shown and/or one or more ofthe blocks of the example process 700 may not be performed.

The egress cell fifo 215A retrieves first cells via a read port of anon-chip MMU 214A (702). The egress cell fifo 215A queues the first cellsfrom the on-chip MMU 214A in a first queue 505A (704). The egress cellfifo 215A receives second cells from a first off-chip MMU 214B via theon-chip transit fifo 212A and a first off-chip transit fifo 212Bassociated with the first off-chip MMU 214B (706). The egress cell fifo215A queues the second cells from the first off-chip MMU 214B in asecond queue 505B (708). The egress cell fifo 215A receives third cellsfrom a second off-chip MMU 314 via the on-chip transit fifo 212A and asecond off-chip transit fifo 312 associated with the second off-chip MMU314 (710). The egress cell fifo 215A queues the third cells from thesecond off-chip MMU 314 in a third queue 505C (712). The scheduler 510of the egress cell fifo 215A then schedules the first, second, and thirdcells for transmission via the on-chip egress processor 218A, theon-chip port macro 217A, and one or more coupled ports (714).

FIG. 8 conceptually illustrates an example electronic system 800 withwhich one or more implementations of the subject technology can beimplemented. The electronic system 800, for example, may be, or mayinclude, one or more of the electronic device 102, 104, 106, the switchdevice 110, a routing device, a desktop computer, a laptop computer, atablet device, a phone, and/or generally any electronic device. Such anelectronic system 800 includes various types of computer readable mediaand interfaces for various other types of computer readable media. Theelectronic system 800 includes a bus 808, one or more processing unit(s)812, a system memory 804, a read-only memory (ROM) 810, a permanentstorage device 802, an input device interface 814, an output deviceinterface 806, one or more network interface(s) 816, and/or subsets andvariations thereof.

The bus 808 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 800. In one or more implementations, the bus 808communicatively connects the one or more processing unit(s) 812 with theROM 810, the system memory 804, and the permanent storage device 802.From these various memory units, the one or more processing unit(s) 812retrieves instructions to execute and data to process in order toexecute the processes of the subject disclosure. The one or moreprocessing unit(s) 812 can be a single processor or a multi-coreprocessor in different implementations.

The ROM 810 stores static data and instructions that are utilized by theone or more processing unit(s) 812 and other modules of the electronicsystem 800. The permanent storage device 802, on the other hand, may bea read-and-write memory device. The permanent storage device 802 may bea non-volatile memory unit that stores instructions and data even whenthe electronic system 800 is off. In one or more implementations, amass-storage device (such as a magnetic or optical disk and itscorresponding disk drive) may be used as the permanent storage device802.

In one or more implementations, a removable storage device (such as afloppy disk, flash drive, and its corresponding disk drive) may be usedas the permanent storage device 802. Like the permanent storage device802, the system memory 804 may be a read-and-write memory device.However, unlike the permanent storage device 802, the system memory 804may be a volatile read-and-write memory, such as random access memory(RAM). The system memory 804 may store one or more of the instructionsand/or data that the one or more processing unit(s) 812 may utilize atruntime. In one or more implementations, the processes of the subjectdisclosure are stored in the system memory 804, the permanent storagedevice 802, and/or the ROM 810. From these various memory units, the oneor more processing unit(s) 812 retrieve instructions to execute and datato process in order to execute the processes of one or moreimplementations.

The bus 808 also connects to the input and output device interfaces 814and 806. The input device interface 814 enables a user to communicateinformation and select commands to the electronic system 800. Inputdevices that may be used with the input device interface 814 mayinclude, for example, alphanumeric keyboards and pointing devices (alsocalled “cursor control devices”). The output device interface 806 mayenable, for example, the display of images generated by the electronicsystem 800. Output devices that may be used with the output deviceinterface 806 may include, for example, printers and display devices,such as a liquid crystal display (LCD), a light emitting diode (LED)display, an organic light emitting diode (OLED) display, a flexibledisplay, a flat panel display, a solid state display, a projector, orany other device for outputting information. One or more implementationsmay include devices that function as both input and output devices, suchas a touchscreen. In these implementations, feedback provided to theuser can be any form of sensory feedback, such as visual feedback,auditory feedback, or tactile feedback; and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

As shown in FIG. 8, bus 808 also couples electronic system 800 to one ormore networks (not shown) through one or more network interface(s) 816.The one or more network interface(s) may include an optical interface,an Ethernet interface, a wireless interface, a multimedia over coaxalliance (MoCA) interface, a reduced gigabit media independent interface(RGMII), or generally any interface for connecting to a network. In thismanner, electronic system 800 can be a part of one or more networks ofcomputers (such as a local area network (LAN), a wide area network(WAN), or an Intranet, or a network of networks, such as the Internet.Any or all components of electronic system 800 can be used inconjunction with the subject disclosure.

Implementations within the scope of the present disclosure can bepartially or entirely realized using a tangible computer-readablestorage medium (or multiple tangible computer-readable storage media ofone or more types) encoding one or more instructions. The tangiblecomputer-readable storage medium also can be non-transitory in nature.

The computer-readable storage medium can be any storage medium that canbe read, written, or otherwise accessed by a general purpose or specialpurpose computing device, including any processing electronics and/orprocessing circuitry capable of executing instructions. For example,without limitation, the computer-readable medium can include anyvolatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM,and TTRAM. The computer-readable medium also can include anynon-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM,NVRAM, flash, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM,NRAM, racetrack memory, FJG, and Millipede memory.

Further, the computer-readable storage medium can include anynon-semiconductor memory, such as optical disk storage, magnetic diskstorage, magnetic tape, other magnetic storage devices, or any othermedium capable of storing one or more instructions. In one or moreimplementations, the tangible computer-readable storage medium can bedirectly coupled to a computing device, while in other implementations,the tangible computer-readable storage medium can be indirectly coupledto a computing device, e.g., via one or more wired connections, one ormore wireless connections, or any combination thereof.

Instructions can be directly executable or can be used to developexecutable instructions. For example, instructions can be realized asexecutable or non-executable machine code or as instructions in ahigh-level language that can be compiled to produce executable ornon-executable machine code. Further, instructions also can be realizedas or can include data. Computer-executable instructions also can beorganized in any format, including routines, subroutines, programs, datastructures, objects, modules, applications, applets, functions, etc. Asrecognized by those of skill in the art, details including, but notlimited to, the number, structure, sequence, and organization ofinstructions can vary significantly without varying the underlyinglogic, function, processing, and output.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, one or more implementationsare performed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In one or more implementations, such integrated circuitsexecute instructions that are stored on the circuit itself.

Those of skill in the art would appreciate that the various illustrativeblocks, modules, elements, components, methods, and algorithms describedherein may be implemented as electronic hardware, computer software, orcombinations of both. To illustrate this interchangeability of hardwareand software, various illustrative blocks, modules, elements,components, methods, and algorithms have been described above generallyin terms of their functionality. Whether such functionality isimplemented as hardware or software depends upon the particularapplication and design constraints imposed on the overall system.Skilled artisans may implement the described functionality in varyingways for each particular application. Various components and blocks maybe arranged differently (e.g., arranged in a different order, orpartitioned in a different way) all without departing from the scope ofthe subject technology.

It is understood that any specific order or hierarchy of blocks in theprocesses disclosed is an illustration of example approaches. Based upondesign preferences, it is understood that the specific order orhierarchy of blocks in the processes may be rearranged, or that allillustrated blocks be performed. Any of the blocks may be performedsimultaneously. In one or more implementations, multitasking andparallel processing may be advantageous. Moreover, the separation ofvarious system components in the embodiments described above should notbe understood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

As used in this specification and any claims of this application, theterms “base station”, “receiver”, “computer”, “server”, “processor”, and“memory” all refer to electronic or other technological devices. Theseterms exclude people or groups of people. For the purposes of thespecification, the terms “display” or “displaying” means displaying onan electronic device.

As used herein, the phrase “at least one of” preceding a series ofitems, with the term “and” or “or” to separate any of the items,modifies the list as a whole, rather than each member of the list (e.g.,each item). The phrase “at least one of” does not require selection ofat least one of each item listed; rather, the phrase allows a meaningthat includes at least one of any one of the items, and/or at least oneof any combination of the items, and/or at least one of each of theitems. By way of example, the phrases “at least one of A, B, and C” or“at least one of A, B, or C” each refer to only A, only B, or only C;any combination of A, B, and C; and/or at least one of each of A, B, andC.

The predicate words “configured to”, “operable to”, and “programmed to”do not imply any particular tangible or intangible modification of asubject, but, rather, are intended to be used interchangeably. In one ormore implementations, a processor configured to monitor and control anoperation or a component may also mean the processor being programmed tomonitor and control the operation or the processor being operable tomonitor and control the operation. Likewise, a processor configured toexecute code can be construed as a processor programmed to execute codeor operable to execute code.

Phrases such as an aspect, the aspect, another aspect, some aspects, oneor more aspects, an implementation, the implementation, anotherimplementation, some implementations, one or more implementations, anembodiment, the embodiment, another embodiment, some embodiments, one ormore embodiments, a configuration, the configuration, anotherconfiguration, some configurations, one or more configurations, thesubject technology, the disclosure, the present disclosure, othervariations thereof and alike are for convenience and do not imply that adisclosure relating to such phrase(s) is essential to the subjecttechnology or that such disclosure applies to all configurations of thesubject technology. A disclosure relating to such phrase(s) may apply toall configurations, or one or more configurations. A disclosure relatingto such phrase(s) may provide one or more examples. A phrase such as anaspect or some aspects may refer to one or more aspects and vice versa,and this applies similarly to other foregoing phrases.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any embodiment described herein as“exemplary” or as an “example” is not necessarily to be construed aspreferred or advantageous over other embodiments. Furthermore, to theextent that the term “include,” “have,” or the like is used in thedescription or the claims, such term is intended to be inclusive in amanner similar to the term “comprise” as “comprise” is interpreted whenemployed as a transitional word in a claim.

All structural and functional equivalents to the elements of the variousaspects described throughout this disclosure that are known or latercome to be known to those of ordinary skill in the art are expresslyincorporated herein by reference and are intended to be encompassed bythe claims. Moreover, nothing disclosed herein is intended to bededicated to the public regardless of whether such disclosure isexplicitly recited in the claims. No claim element is to be construedunder the provisions of 35 U.S.C. §112, sixth paragraph, unless theelement is expressly recited using the phrase “means for” or, in thecase of a method claim, the element is recited using the phrase “stepfor.”

The previous description is provided to enable any person skilled in theart to practice the various aspects described herein. Variousmodifications to these aspects will be readily apparent to those skilledin the art, and the generic principles defined herein may be applied toother aspects. Thus, the claims are not intended to be limited to theaspects shown herein, but are to be accorded the full scope consistentwith the language claims, wherein reference to an element in thesingular is not intended to mean “one and only one” unless specificallyso stated, but rather “one or more.” Unless specifically statedotherwise, the term “some” refers to one or more. Pronouns in themasculine (e.g., his) include the feminine and neuter gender (e.g., herand its) and vice versa. Headings and subheadings, if any, are used forconvenience only and do not limit the subject disclosure.

What is claimed is:
 1. A device comprising: a memory management unitconfigured to buffer first cells for transmission; an egress cellcircuit comprising at least a first queue and a second queue, the egresscell circuit configured to: queue the first cells from the memorymanagement unit in the first queue; queue second cells from anothermemory management unit of another device in the second queue; andschedule the first cells from the first queue and second cells from thesecond queue for transmission via an egress processor; and the egressprocessor configured to transmit the first and second cells over atleast one first port.
 2. The device of claim 1, further comprising atransit circuit that is communicatively coupled to another transitcircuit of the another device, the transit circuit configured to:receive the second cells from the another memory management unit of theanother device via the another transit circuit without the second cellspassing through another egress processor of the another device; and passthe second cells directly to the egress cell circuit bypassing thememory management unit of the device.
 3. The device of claim 2, whereinthe transit circuit comprises a receive queue, wherein the transitcircuit is further configured to queue the second cells in the receivequeue prior to passing the second cells directly to the egress cellcircuit.
 4. The device of claim 2, further comprising: an ingressprocessor configured to receive the first cells and provide the firstcells to the memory management unit.
 5. The device of claim 4, whereinthe transit circuit is further configured to: receive a request for atleast some of the first cells from the another transit circuit of theanother device; retrieve the at least some of the first cells from thememory management unit; and transmit the at least some of the firstcells to the another transit circuit of the another device fortransmission via another egress processor over at least one second portof the another device, the at least some of the first cells not beingtransmitted via the egress processor of the device over the at least onefirst port.
 6. The device of claim 4, wherein the transit circuit of thedevice is directly coupled to the another transit circuit of the anotherdevice via a first link, and the transit circuit of the device isseparately indirectly coupled to the another transit circuit of theanother device via a second link through an intermediary transit circuitof an intermediate device.
 7. The device of claim 6, wherein the transitcircuit comprises a scheduler configured to: receive first linkinformation for the first link and second link information for thesecond link; determine one of the first or second link to transmit theat least some of the first cells to the transit circuit of the anotherdevice based at least in part on the first link information and thesecond link information; and transmit the at least some of the firstcells to the another device via the determined first or second link. 8.The device of claim 7, wherein the device, the another device, and theintermediate device are each a separate integrated circuit.
 9. Thedevice of claim 8, wherein the separate integrated circuits are mountedon a common substrate and the transit circuit of the device iscommunicatively coupled to the another transit circuit of the anotherdevice via a serializer-deserializer interface.
 10. The device of claim4, wherein the ingress processor is further configured to: receive thirdcells and transmit the third cells directly to the egress cell circuitbypassing the memory management unit; and the egress cell circuit isfurther configured to schedule the third cells for transmission via theegress processor in conjunction with scheduling the first and secondcells for transmission via the egress processor.
 11. The device of claim2, wherein the egress cell circuit comprises a scheduler configured toschedule the first cells from the first queue and the second cells fromthe second queue for transmission via the egress processor.
 12. A devicecomprising: an ingress processor configured to receive first cells andprovide the first cells to a memory management unit; the memorymanagement unit configured to buffer the first cells for transmission;an egress processor configured to transmit at least a portion of thefirst cells over at least one first port; a transit circuit that iscommunicatively coupled to another transit circuit of another device,the transit circuit configured to: receive a request for at least someof the first cells from the another transit circuit of the anotherdevice; retrieve the at least some of the first cells from the memorymanagement unit via a read port of the memory management unit; andtransmit the at least some of the first cells to the another transitcircuit of the another device for transmission via another egressprocessor over at least one second port of the another device, the atleast some of the first cells bypassing the egress processor of thedevice.
 13. The device of claim 12, further comprising: an egress cellcircuit comprising at least a first queue and a second queue, the egresscell circuit configured to: queue, in the first queue, the at least theportion of the first cells retrieved from the read port of the memorymanagement unit; queue, in the second queue, second cells retrieved fromanother memory management unit of the another device via the transitcircuit; and schedule the at least the portion of the first cells fromthe first queue and the second cells from the second queue fortransmission via the egress processor.
 14. The device of claim 13,wherein the egress processor is further configured to transmit the atleast the portion of the first cells and the second cells over the atleast one first port.
 15. The device of claim 13, wherein the transitcircuit is further configured to: receive a request for the second cellsfrom the egress cell circuit; request the second cells from the anothertransit circuit of the another device; receive the second cells from theanother memory management unit of the another device via the anothertransit circuit of the another device; and provide the second cells tothe egress cell circuit bypassing the memory management unit.
 16. Amethod comprising: retrieving first cells from an on-chip memorymanagement unit, the first cells having been received via an on-chipingress processor; receiving second cells from a first off-chip memorymanagement unit and third cells from a second off-chip memory managementunit, the second cells having been received via a first off-chip ingressprocessor and the third cells having been received by a second off-chipingress processor; scheduling the first, second, and third cells fortransmission via an on-chip egress processor without buffering thesecond or third cells in the on-chip memory management unit; andtransmitting the first, second, and third cells over at least one portvia the on-chip egress processor.
 17. The method of claim 16, wherein afirst portion of the second cells are retrieved from the first off-chipmemory management unit via a first link, the third cells are retrievedfrom the second off-chip memory management unit via the first link, anda remaining portion of the second cells are retrieved from the firstoff-chip memory management unit via a second link that is physicallydistinct from the first link.
 18. The method of claim 16, wherein thesecond cells retrieved from the first off-chip memory management unitbypass a first off-chip egress processor associated with the firstoff-chip memory management unit and the third cells retrieved from thesecond off-chip memory management unit bypass a second off-chip egressprocessor associated with the second off-chip memory management unit.19. The method of claim 16, further comprising: queuing the first,second, and third cells in first, second, and third queues,respectively, prior to scheduling the first, second, and third cells fortransmission, wherein the first, second, and third queues are separatefrom the on-chip memory management unit as well as the first and secondoff-chip memory management units.
 20. The method of claim 16, furthercomprising: receiving fourth cells directly from the on-chip ingressprocessor, the fourth cells bypassing the on-chip memory managementunit; and scheduling the fourth cells for transmission via the on-chipegress processor without buffering the fourth cells in the on-chipmemory management unit.